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(57) Abstract 

The present invention has application to 
multiple listener digital systems. A central 
processing unit (CPU) is coupled to a mass 
storage device, such as for example, a read- 
only memory multimedia cartridge, CD-ROM, 
floppy disk or hard disk drive. The mass storage 
device stores an application audio program 
which is executed by the CPU and further 
stores a library of sound data. A sound 
synthesizer circuit is coupled to the CPU for 
generating digital audio signals from the sound 
data stored in said mass storage device. Under 
the control of a single executable program, 
the CPU selectively provides sound data for 
a listener of the audio program to the sound 
synthesizer circuit. A digital to audio converter 
(DAC) is coupled to receive the digital audio 
signals from the sound synthesizer circuit and 
convert the digital audio signals into analog 
audio signals. A listening device, such as a 
headphone, is coupled to the DAC to convert 
the analog audio signals into sound waves to be 
heard by the listener. Under program control 
the CPU directs certain of the sound data to 
each of the listeners, such that the listeners hear 
separate and independent audio as required by 
the audio program. 
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METHOD AND APPARATUS FOR PROVIDING INDEPENDENT 
AUDIO IN MULTIPLE LISTENER DIGITAL SOUND SYSTEMS 

1. FIELD OF THE INVENTION • 

The present invention relates to the field of digital audio systems, and more 
particularly, to audio systems used in digital devices such as personal computers, 
workstations, network servers and dedicated multimedia game devices and any other 
digital device where it may be desirable to provide separate audio to individual listeners. 

2. ART BACKGROUND - 

Historically, multimedia game systems utilized two dimensional landscapes and 
iconic bit mapped action characters. Game players viewed this two dimensional world 
through a low resolution cathode ray tube (CRT) which typically comprised a home 
television set. Software content was limited by low resolution graphics, low processor 
speeds, and in some instances, monaural audio provided to the game players. 

Recent innovations to both home cartridge-based game systems and game 
programs executed by personal computers (PCs) have changed many of the systems" 
characteristics. Today, richly-textured, three dimensional graphics supported by 
hardware chip sets supplement or replace older bit mapped two dimensional graphics. 
CD-ROM players have expanded storage capacity and the addition of modems for 
network game playing expands the number and geographical sphere of players. These 
recent innovations have effectively eliminated the traditional boundaries between games 
viewed on televisions and those viewed on personal computers. 

However, although the sophistication of game programs and the displayed 
graphics have markedly improved from earlier generations of cartridge and PC-based 
systems, audio quality has generally lagged. More particularly, prior art monaural or 
dedicated game devices and personal computer systems typically provide a single 
monaural or stereo output to all of the players of the game simultaneously. Thus, each of 
the players of a multimedia game perceive the same audio experience. Although the audio 
may be stereophonic and 16 bit CD quality, it does not vary between the players. 

As will be described, the present invention provides an enhanced audio experience 
for each of the players of a multimedia game heretofore unknown in the prior art. By 
providing separate and independent audio to each of the players of either a dedicated game 
machine or personal computer, the game experience is enhanced and tailored to the 
particular player. The present invention permits separate and independent audio to be 
perceived by each player, or groups of players, which may be a function of the game, of 
the players' language, the players' ability to handle more complex audio feedback, and/or 
the players' technical understanding. 

1 
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It will be appreciated that the present invention is useful for a wide variety of 
applications apart from games. Multiple user application programs, such as for 
education, may prove more effective if the individual users may be provided with 
individualized sound tracks and the present invention may be employed in all such 
applications. It will be further appreciated that the present invention may be used in 
conjunction with non-interactive application programs. Indeed, the present invention may 
be employed with any type of digital device where it is desirable to provide separate and 
independent sound to individual listeners or groups of listeners. 

SUMMARY OF THE TNVF.NTTDM 
The present invention has application to any digital system such as a personal 
computer, dedicated multimedia game system, network server, workstation and the like 
wherein a single executable program at least partly controls the routing of separate and 
independent sound data to particular listeners. A central processing unit (CPU) may be 
coupled to a mass storage device, such as for example, a read-only memory game 
cartridge, CD-ROM, floppy disk or hard disk drive or remote network server. The mass 
storage device stores a program (hereinafter referred to as an "audio program") that 
includes audio components which is executed by the CPU and may also store a library of 
sound data. It will be appreciated that the CPU may be coupled with any type of sound 
source such as a real time sound source. In a preferred embodiment, a sound synthesizer 
circuit is coupled to the CPU for generating digital audio signals from the sound data 
stored in said mass storage device. Under program control, the CPU selectively provides 
sound data for a listener of the audio program to the sound synthesizer circuit. A digital 
to audio converter (DAC) is coupled to receive digital audio signals from the sound 
synthesizer circuit and convert the digital audio signals into analog audio signals. In the 
present embodiment, a listening device, such as a headphone, is coupled to the DAC to 
convert the analog audio signals into sound waves to be heard by the listener. Under 
program control the CPU directs certain portions of the sound data to each of the 
listeners, such that the listeners hear separate and independent audio as required by the 
program. 

In a first embodiment, the DAC provides a stereo output having left and right 
channels. The CPU directs sound data for a first listener to the left channel and sound 
data for a second listener to the right channel. The two listeners each control separate 
multimedia controllers and wear headphones coupled to the multimedia controllers. The 
headphones for the first listener are coupled to the left output channel of the DAC and the 
headphone for the second listener are coupled to the right output of the DAC. Thus, each 
listener hears a separate and independent monaural channel of audio. 
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In a second embodiment, the stereo output of the DAC is coupled to an analog 
multiplexer which is under the control of the CPU. A plurality of listeners operate \ 
multimedia controllers which are coupled to the multiplexer. The CPU directs sound data 
destined for each listener independent of sound data destined for other listeners. The i 
CPU controls the multiplexer selection such that the appropriate multimedia controller is \ 
selected to receive the desired audio at the appropriate time. As in the first embodiment, 
the listener's headphones are coupled to the multimedia controller such that the listener 
hears audio intended for that listener independent of the other listeners. 

In a third embodiment, the sound synthesizer circuit is coupled to a sound bus. 
Separate DACs are provided for each hand controller and are coupled between the audio 
bus and the hand controllers. A DAC control bus is provided which is coupled to the 
CPU and enable circuits for each of the DACs. In operation, the CPU executing the 
audio program provides sound data to the sound synthesizer circuit which provides its J 
digital audio signal output to the audio bus. Each of the DACs are selectively enabled by a 
control signal coupled over the control bus by the CPU, such that substantially concurrent 
with the sound synthesizer circuit providing digital audio signals to the sound bus, the 
CPU enables the appropriate DAC corresponding to the desired destination multimedia 
controller and headphone. 

In a fourth embodiment, the sound synthesizer circuit is coupled to a digital input 
device controller which is in turn coupled to a bi-directional high speed digital bus. Each 
multimedia controller is coupled to the bus and includes a microcontroller and interface 
logic to receive and send data over the bus. The microcontroller is coupled to a DAC to 
which a listener's headphones are in turn coupled. In operation, an audio program 
executed by the CPU utilizes sound data stored in the mass storage device. In accordance 
with program commands, the CPU forwards sound data to a particular multimedia ' 
listener operating one of a plurality of multimedia controllers coupled to the bus. The 
sound synthesizer circuit converts the sound data into digital audio signals, which are 
encapsulated into packets by the digital input device controller in accordance with the 
particular protocol of the bus. The packet includes address information unique to the 
destination multimedia controller. The packet is received by the destination multimedia 
controller and decapsulated to recover the digital audio signals which are then coupled to 
the DAC. The DAC converts the digital audio signals into analog audio signals which are 
provided to the listener's headphones. 



BRIEF DESCRIPTI ON OF THE DRAWINGS 
Figure 1 is a block diagram illustrating a first embodiment of the present invention 
wherein monaural audio is provided to each listener. 
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Figure 2 is a block diagram of a second embodiment of the present invention in 
which stereo sound is provided to each listener under the control of a central processing 
unit. 

Figure 3 is a block diagram illustrating a third embodiment of the present 
invention wherein multiple digital to audio converters (DACs) are coupled to an audio 
bus, and wherein each of the DACs are under the control of a central processing unit. 

Figure 4 illustrates a block diagram of a fourth embodiment of the present 
invention wherein each of the listeners* controllers are coupled to a high speed digital 
bus. 

Figure 5 is a block diagram of a multimedia controller utilized by the present 
invention's fourth embodiment shown in Figure 4. 

DETAILED DESC RIPTION OF THE INVENTION 
The present invention discloses methods and apparatus for providing separate and 
independent audio experiences to persons listening to the audio output of a digital system. 
In the following description, numerous specific details are set forth, such as system 
architectures, representative devices, and exemplary hardware configurations, etc. to 
provide a thorough understanding of the present invention. However, it will be apparent 
to one skilled in the art that the present invention may be practiced without these specific 
details. 

In other instances, well-known circuits and structures are not described in detail in 
order not to obscure the present invention unnecessarily. For example, since the present 
invention relates to the audio component of a digital system, various video, control and 
other subsystems which are common to various types of digital systems are not shown in 
the figures or described. It will be appreciated by one skilled in the art that these 
additional subsystems for video, video compression, system integration and control may 
be necessary to integrate the present invention into a particular digital system although the 
present invention is not limited to digital systems including video subsystems. Similarly, 
power, control and other lines and devices are not shown in the drawings, but their 
addition will be apparent to one skilled in the art based upon the description of the present 
invention in this Specification. Moreover, although the present invention is described 
with reference to a personal computer, it should be noted that the teachings of the present 
invention may be dedicated multimedia systems, other general purpose digital computers, 
DVD players (digital video disk players), set top boxes and the like, which are configured 
by either hardware or software to run programs that include, either directly or indirectly, 
sound information. 
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As used in this specification, the term "multimedia controller" includes human 
input devices such as hand-held game controllers, joysticks, remote controls, keyboard 
and similar devices. 

Referring now to Figure 1, a first embodiment of the present invention is 
illustrated. A central processing unit (CPU) 10 runs a single instance of an executable 
program that at least partly controls the routing of separate and independent sound data to 
particular listeners. The present invention is thus in contrast to network games such as 
"Marathon" where multiple copies of an executable program run on separate computers 
and each copy of the program provides sound data to a user through his or her associated 
computer. Although, according to the present invention, a single executable program, as 
opposed to multiple copies of the same program, provides separate sound to listeners, it 
will be appreciated that the single program may be run across multiple processors within a 
single digital device or within a network. For example, a single computer may use 
multiple processors to execute a main program. For another example, a server may 
provide distinct audio programs for separate users of a game, movie or such. Each user 
may have a separate computer that translates, extracts or otherwise processes the audio 
program for that user but the primary program is running on the server. 

The CPU 10 is coupled through supporting circuitry (not shown) to an 
input/output (I/O) circuit 12. The I/O circuit 12 is used to communicate information in 
appropriately structured form to I/O devices and to provide a video output to a display 14. 
Where the present invention is applied to a personal computer, the display 14 may 
comprise a computer monitor. As shown, the visual output of an audio program, such as 
for example, a multimedia game, an educational program or the like is displayed on the 
display 14. Although Figure 1 illustrates a multimedia system, it will be understood that 
the present invention is applicable to digital systems that do not have video components. 
A mass data storage device 16 is coupled to the CPU 10. The mass data storage device 
16 may comprise a hard disk drive, a read-only memory (ROM) multimedia cartridge, a 
CD-ROM, remote network server or other read-only or writable mass storage device. 
The mass storage device 16 stores the application program which is executed by CPU 10. 
As such, the CPU 10 is under program control in accordance with the instructions stored 
in the mass storage device 16. 

Mass storage device 16 further stores a library of sound data which is used by the 
particular program executed by CPU 10. The sound data stored on the mass storage 
device 16 may comprise sound data, analog sound data, MIDI code sequences and the 
like. Therefore, it is contemplated that whatever form mass storage device 16 takes, the 
necessary application program, images (if any), and sounds required by the particular 
application are stored therein and are accessible to the CPU 10. It will further be 
appreciated that in the embodiments disclosed in this Specification, that the application 
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program, and/or the sound data, may be downloaded from a network for storage in the 
mass storage device 16 or for direct execution by the CPU 10. It will be further 
appreciated that in the embodiments the CPU 10 need not be coupled to a mass storage 
device that stores sound data and may be coupled with any type of sound source such as a 
real time sound source which may be natural or synthetic. 

As illustrated in Figure 1, a sound synthesizer circuit 18 is coupled to, and under 
the control of, CPU 10. In practice, the sound synthesis circuit 18 may comprise digital 
hardware or be implemented in software to achieve substantially the same functionality. 
The sound synthesizer circuit generates the required audio output under the control of 
CPU 10 utilizing the sound data stored in the mass storage device 16. The output of the 
sound synthesizer circuit 18 is coupled to a digital to audio converter (DAC) 20, which 
receives the digital output of the sound synthesizer circuit 18 and converts the digital 
audio data into analog electrical signals. 

As shown in Figure 1, the output of the DAC 20 comprises left (L) output 22 and 
right (R) output 24. In prior art systems, the left 22 and right 24 outputs comprise a 
stereo audio output which was perceived by all listeners of the multimedia system. 
However, in accordance with the teachings of the present invention, the left 22 and right 
24 outputs of the DAC 20 are coupled to separate multimedia controllers 26 and 28, as 
shown in the figure. As is well-known, multimedia controllers 26 and 28 are used by 
listeners of the multimedia application to interact with CPU 10 through the I/O circuit 12. 
Depending on the particular multimedia application program executed by CPU 10, each of 
the respective listeners must provide appropriate inputs using multimedia controllers 26 
and 28. As illustrated, a headphone 29 is coupled to the multimedia controller 26, and a 
headphone 32 coupled to the multimedia controller 28. In the embodiments disclosed in 
this Specification, it will be appreciated that the separate sound data need not be provided 
through a headphone but may be provided, for example, to separate groups of listeners in 
different rooms. It will further be appreciated that, as previously mentioned, the present 
invention is in no way limited to interactive applications or programs that include a video 
component. 

In accordance with the teachings of the present invention, rather than providing a 
stereophonic audio output to both listeners, this first embodiment provides independent 
audio outputs in monaural form to each of the respective listeners. Utilizing the teachings 
of the present invention, it is contemplated that the sound data stored in the mass storage 
device 16 will be selectively provided, that is routed, to each of the listeners by CPU 10 
as required by the particular audio program. Thus, the left and right channels output 
different audio experiences to the respective listeners. 

For example, the application may comprise a program that allows two or more 
users to simultaneously create a multimedia application. One user may be assigned the 
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task of creating shape outlines while the other user may be assigned the task of filling in 
the outlines. The users may be provided with audio feedback to indicate certain events, 
such as when a boundary has been inappropriately crossed. To avoid confusion, it 
would be desirable to provide the users with separate audio so that each user may 
concentrate on his or her own creation process. 

Thus, a user operating multimedia controller 26 would hear sounds unique to his 
or her own creations. Similarly, a user operating multimedia controller 28 would hear 
sounds unique to his or her own creations. Each user would not necessarily hear the 
audio provided to the other user, thereby avoiding confusion, although in certain 
applications it may be desirable for both users to hear some common sounds. These 
common sounds may be supplied under the control or direction of CPU 10, or 
alternatively, may be provided using, a separate line, or using a sum or difference 
technique known in the Art. The invention also contemplates providing two or more 
channels of sound in each output so each user may experience stereo or multichannel 
sound. 

Continuing to refer to Figure 1, it will be noted that both headsets 29 and 32 are 
illustrated as being coupled to the multimedia controllers 26 and 28, respectively, through 
a wire coupling. It will be appreciated by one skilled in the art that in the embodiment of 
Figure 1, as well as in the other embodiments disclosed in the Specification, both the 
multimedia controllers and headphones may be coupled to the multimedia system in a 
wireless manner. For example, it is contemplated that the multimedia controllers may 
communicate with the multimedia system, and/or the headphones may receive audio 
signals, either from the multimedia system or from the multimedia controllers using, for 
example, infrared or RF transmission mediums. 

Referring now to Figure 2, therein is disclosed a block diagram illustrating a 
second embodiment of the present invention. As in the embodiment shown in Figure 1, a 
CPU 40 is coupled to an I/O circuit 42. The I/O circuit 42 is coupled to a display 44 for 
displaying the visual output of an application program executed by the CPU 40. A mass 
storage device 46 stores the audio program for execution by the CPU 40 as well as sound 
data to provide the audio experience for the application program. A sound synthesizer 
circuit 50 is coupled to the CPU 40 for generating a digital representation of the required 
sounds under the control of the CPU 40, and utilizing the sound data stored in the mass 
storage device 46. 

As shown, a DAC 52 is coupled to the sound synthesizer circuit 50 for converting 
the digital output of the sound synthesizer circuit 50 into an analog output. An analog 
multiplexer 55 is coupled to the output of the DAC 52. In addition, a multiplex control 
line 56 is provided between the CPU 40 and the analog multiplexer 55. The analog 
multiplexer 55 is coupled to multimedia controllers 60, 62, 64 and 66. Headphones 68, 
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70, 72 and 74 are coupled to the respective multimedia controllers as shown in the figure. 
Although the analog multiplexer 55 is illustrated as a four-way multiplexer in Figure 2, it 
will appreciated by one skilled in the art that the multiplexer 55 may have fewer or greater 
number of multimedia controllers coupled to it as required by a particular multimedia 
system or application. The multimedia controllers illustrated in Figure 2 are coupled to 
the CPU 40 over a multimedia control bus 80 through the I/O circuit 42. It will be 
appreciated that the multimedia control bus 80 may comprise an analog parallel bus or, 
with appropriate circuitry and supporting logic, a digital serial or parallel bus. In 
addition, although Figure 2 illustrates the multiplex control line 56 coupled between the 
CPU 40 and the analog multiplexer 55, it will be noted by one skilled in the art that 
additional circuitry may be required (and is not shown) in order to provide control over 
the analog multiplexer 55 by the CPU 40. However, the additional support circuitry 
required to implement the present invention as illustrated in Figure 2 is well-known, and 
therefore, not described in this Specification. 

In operation, separate and independent audio experiences are provided to each of 
the users operating the multimedia controllers 60, 62, 64 and 66 by multiplexing the 
audio output from the DAC 52. For example, audio to be routed to a user operating 
multimedia controller 60 is provided by the CPU 40 to the sound synthesis circuit 50 and 
the DAC 52. The output of the DAC 52 is coupled to the analog multiplexer 55 which is 
selected by the CPU 40 to provide its output at the appropriate time to the multimedia 
controller 60 and headphones 68. 

Although not shown in the figure, it may be desirable depending on the speed of 
the audio output to buffer the output of the analog multiplexer 50, or sound synthesis 
circuit 52, to achieve proper timing and realism in multimedia playback. In addition, in 
certain applications, it may be desirable to group multiple users into teams which receive 
the same audio experience. For example, in the embodiment illustrated in Figure 2 the 
multimedia application may group users operating the multimedia controllers 60 and 62 as 
a first team, and users operating the multimedia controllers 64 and 66 as a second team. 
As such, the analog multiplexer 55 need only switch between two groups of users 
thereby reducing the speed requirements of the system. In multimedia applications, such 
as for example a helicopter gun ship squadron battling a tank squadron, the users 
operating within each group of multimedia controllers would hear only audio signals 
appropriate for their group. Thus, the users commanding the helicopter gun ships will 
hear only helicopter gun ship audio, and the tank users will hear only tank 
communications and other audio associated with the tank group. Of course, it will be 
understood that there may be instances where it is desirable to provide some common 
audio to all users. In such event, either a separate common line may be used between the 
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multiplexer and the multimedia controllers, or alternatively, the multiplexer may 
sequentially provide the common audio to each multimedia controller. 

Referring now to Figure 3, a block diagram is provided illustrating a third 
embodiment of the present invention. As in the embodiments of Figures 1 and 2, a CPU 
100 is coupled to an I/O circuit 102. The I/O circuit 102 is coupled to a display 104 for 
displaying the application program executed by the CPU 100 under the control of 
software stored in, for example, a mass storage device 106. The mass storage device 106 
further stores sound data required by the particular audio program executed by the CPU 
100. 

As shown, a sound synthesizer circuit 108 is coupled to the CPU 100 for ✓ 
generating digital representations of sounds required by the program using, for example, 
the audio data stored in the mass storage device 106. The sound synthesizer circuit 108 is 
coupled to an audio bus 1 10. DACs 1 12, 1 14, 1 16 and 1 18 are coupled to receive the 
output of the sound synthesizer circuit 108 over the audio bus 1 10. Although four DACs 
are illustrated in Figure 3, it will appreciated that depending upon the particular 
multimedia system and/or program, any number of DACs may be coupled to sound bus 
110. 

As illustrated, multimedia controllers 120, 122, 124 and 126 are coupled, 
respectively, to the DACs 1 12, 1 14, 1 16 and 1 18. Headphones 130, 132, 134 and 136 
are coupled to the respective multimedia controllers. A multimedia control bus 140 is 
provided to couple signals originating from each of the multimedia controllers to the CPU 
100 through the I/O circuit 102. The multimedia control bus 140 may comprise an analog 
parallel bus, or alternatively, with appropriate digital logic, a digital serial or parallel bus. 
As shown in Figure 3, DACs 1 12, 1 14, 1 16 and 1 18 are each coupled to a DAC control 
bus 142. The CPU 100 is also coupled to each of the DACs over the DAC control bus 
142. 

In operation, CPU 100 executing the application program stored in the mass 
storage device 106 provides sound data to the sound synthesizer circuit 108. The output 
of the sound synthesizer circuit 108 is coupled to the audio bus 1 10. Sound data destined 
for a particular user is routed to that user by selectively enabling each of the DACs 1 12, * 
1 14, 1 16 and 1 18 by a control signal provided over the DAC control bus 142 from the 
CPU 100. Audio data directed to the user operating, for example, the multimedia 
controller 120 is provided over the audio bus 1 10 by the sound synthesizer circuit 108. 
Substantially concurrent to the sound synthesizer circuit 108 providing audio destined for 
multimedia controller 120 the CPU 100 provides an enable signal to the DAC 1 12 over 
the DAC control bus 142. Since only DAC 1 12 is enabled (DACs 1 14, 1 16 and 1 18 
being disabled) audio data destined for multimedia controller 120 is only received by 
multimedia controller 120 and not the other multimedia controllers coupled to the system. 
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Similarly, audio data destined for any of the other multimedia controllers illustrated in 
Figure 2 is provided to the desired multimedia controller through the selective enabling of 
the corresponding DAC by the CPU 100. Common audio data which is destined for all 
(or groups) of multimedia controllers is provided by enabling all, or a selected set, of the 
DACs. 

It will be appreciated by one skilled in the art that it may be desirable to buffer the 
output of the sound synthesizer circuit 108 to ease the timing requirements for the 
enabling and disabling of the DACs. It is contemplated that the buffering of the DACs 
may be accomplished at a much faster rate than the sound output requirements of the 
multimedia controllers coupled to the system. In this manner, the audio output of each 
multimedia controller would approach that of a real time system, such that the user does 
not perceive any undesirable delays in receiving his audio output, and each listener will 
perceive simultaneous audio experiences despite the serial nature of the output of the 
sound synthesis circuit 108. 

Referring now to Figure 4 in conjunction with Figure 5, there is shown a fourth 
embodiment of the present invention. As in the embodiments illustrated in Figures 1 
through 3, a CPU 200 is coupled to an I/O circuit 202 and to a mass storage device 204. 
The I/O circuit 202 is coupled to a display 206 for displaying the visual portion of an 
application program executed by the CPU 200 and stored in the mass storage device 204. 
20 Sound data representing the audio portion of the program is also stored in the mass . 
storage device 204, as previously described with reference to the other embodiments. 

A sound synthesizer circuit 208 is coupled to the CPU 200 for generating the 
required sound signals utilizing the sound data stored in the mass storage device 204. 

A digital input device controller 210 is coupled to receive the digital audio output 
25 from the sound synthesizer circuit 208. In practice, it is contemplated that the digital 
input device controller may comprise hardware under the control of the CPU 200. The 
digital input device controller 210 is coupled to a digital bi-directional high speed bus 
212. Multimedia controllers 214, 216, 218 and 220 are also coupled to the high speed 
bus 212. As illustrated in Figure 4 and as will be described more fully below, digital 
30 audio is coupled over the high speed bus 2 12 to the multimedia controllers, and listener 
commands are coupled through the digital input device controller 210 to the CPU 200. 
Headphones 222, 224, 226 and 228 are coupled to their respective multimedia controllers 
to provide independent audio to each of the listeners. As will be appreciated, the 
embodiment illustrated in Figure 4 is almost entirely digital. Selections made by users 
using the multimedia controllers 214, 216, 218 and 220 are coupled over the high speed 
bus 212 to the digital input device controller 210 and the CPU 200. Similarly, digitized 
audio to be directed independently to each of the users is coupled over the high speed bus 
212 and addressed to the desired multimedia controller. It is contemplated that the high 

10 



35 



BNBOOCIO; <WO_S7TXVW1 A1 JU> 



WO 97/00491 



PCT/US96/10450 



speed bus 212 may comprise the Apple® Firewire™ (IEEE 1394-1995), the Intel® 
Universal Serial Bus, or other bus having similar high speed capabilities. 

Also shown in Figure 4 are optional microphones 230, 232, 234 and 236 coupled 
to each of the multimedia controllers. The embodiment illustrated in Figure 4 provides 
not only full stereo audio to each of the users, but also permits audio feedback from each 
of the users to the other users as may be desired in a particular multimedia or other 
application. It will be appreciated that other forms of sound, such as monaural sound, 
may be provided to each of the users. 

In operation, a program executed by the CPU 200 utilizes sound data stored in the 
mass storage device 204 to generate unique independent sounds to each listener as 
provided by the application. The digital input device controller 210 handles the two-way 
communication of data and sound between the multimedia controllers 2 14, 2 1 6, 2 1 8 and 
220. Illustrated in Figure 5 is a representative multimedia controller 214 which includes a 
serial bus interface cirr ]40 coupled over line 242 to the high speed bus 212. A 
microcontroller 242 i .pled to the serial bus interface 240. The microcontroller 242 
may comprise a ge r , ai purpose microprocessor, or alternatively, may comprise a custom 
microcontroller. . DAC 246 is coupled to the microcontroller 242 and to headphones * 
248. Sound data is provided from the sound synthesizer circuit 208 through the digital *. 
input device controller 210 and coupled over the high speed bus 212 to the serial bus 
interface 240. The microcontroller 242 further couples this digitized audio to the DAC 
246 which converts the digitized audio into analog signals. A user utilizing the 
multimedia controller 214 will then hear independent audio intended for, and directed to, 
that listener using the headphones 248. 

The multimedia controller 214 includes various input devices such as push button 
switches 260 and four-way rocker switches 262. A listener operating the multimedia 
controller 2 14 provides multimedia input commands to the CPU 200 through the use of 
switches 260 and 262. Signals generated by the switches 260 and 262 are coupled to the 
microcontroller 242 and are ultimately coupled through the serial bus interface 240 to the 
high speed bus 212. The digital input device controller 210 receives these input 
command signals and couples them to the CPU 200. Similarly, for audio input at the 
user level, microphone 230 is coupled to an analog to digital (A/D) converter 270, which 
is in turn coupled to the microcontroller 242. Audio input may thereby be provided to 
other users, or to the CPU 200, depending on the multimedia application. 

Thus, it will be appreciated that the embodiment illustrated in Figures 4 and 5 is 
almost entirely digital in nature and provides separate independent audio to each of the 
users operating the various multimedia controllers. Since each multimedia controller is 
independently addressable, sound data packets, which include an indicator, such as an 
address, that correlates the particular data packet with a particular destination, may be 
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directed independently to the multimedia controllers. Input command signals may be 
coupled back to the CPU 200 by the listener using switches 260 and 262, such that these 
desired input command signals are coupled in a packetized form over the high speed bus 
212 for receipt by the CPU 200. Digital audio signals destined for a particular multimedia 
controller and listener are encapsulated by the digital input device controller 210 in 
accordance with the protocol of the bus 212. The packet is routed to the destination 
multimedia controller and decapsulated to recover the digital audio signals. The digital 
audio signals are then coupled to the DAC which provides analog audio signals to the 
listener's headphones. 

Accordingly, the embodiment illustrated in Figure 4 permits very high levels of 
performance and flexibility. Individuals and groups of listeners may selectively receive 
communications between one another depending upon the application. For example, in 
the context of an educational application wherein different student groups share access to 
the same application, those group members may receive and transmit secure 
communications between one another through the use of headphones (for example, 
headphone 248) and the microphones (for example, microphone 230) coupled to the 
multimedia controllers. In addition, as digital systems become more sophisticated, it is 
contemplated that each of the listeners may communicate through voice recognition 
software directly with the CPU 200, and thereby achieve a level of dialog and interaction 
with the application program. 

It will be appreciated that the embodiments illustrated herein may be networked 
through local area networks (LAN), wide area networks (WAN), telephone, optical fiber, 
two-way cable, and/or satellite links with other multimedia listeners and/or computers. It 
is contemplated that the networking of systems employing the teachings of the present 
invention will permit independent and separate audio experiences to each of the listeners 
receiving sound information through the network. 

Further, the present invention is applicable to any architecture wherein a single 
executable program directs the listening experience for two or more listeners. The single 
executable program may be run across multiple processors, such as where a program 
operating on a first processor invokes a remote procedure call to a second processor. The 
program may also be resident on a server, such as a world wide web server, that provides 
multimedia content to a client. Even though the client may be running an executable 
program to interpret the instructions of the server program, the server program, which 
corresponds to the audio program of the present invention, directs the listening 
experience. 

It is also possible for the server program to merely download data or an 
application program. The client may then interpret the downloaded content (eg. display 
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any visual portions and convert the audio data to sound) or run the application program 
according to the teachings of the present invention. 

As described in this Specification, the present invention may be implemented in 
one of a variety of embodiments depending upon the level of sophistication and technical 
performance required. Although the present invention has been described with reference 
to a few exemplary Figures 1 through 5, it will be apparent that many alternatives, 
modifications and variations may be made in light of the foregoing description. 
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What is claimed is: 

1 1 . A computer game for providing separate and independent audio to a first and 

2 a second user, comprising: 

3 storage means for storing sound data; 

4 processor means operating under the control of a program and coupled to said 

5 storage means, for selecting and retrieving first sound data for said first user and 

6 second sound data for said second user; 

7 sound synthesizer means coupled to receive said first and second sound data 

8 and converting said sound data into first and second digital audio signals, 

9 respectively; 

10 dig"* 1 to aud »o converter (DAC) means for receiving said first and second 

1 1 digital audio signals from said sound synthesizer means and converting said first and 

12 second digital audio signals into corresponding first and second analog audio signals; 

1 3 routing means coupled to receive said first and second analog audio signals 

14 and selectively providing said first analog audio signals to said first user and said 

15 second analog audio signals to said second user; 

16 a listening device coupled to said routing means and provided to said first and 

1 7 second users to convert said first and second analog audio signals to sound waves 

1 8 such that said first user hears said first analog audio signals and said second user 

19 hears said second analog audio signals. 

1 2 . The apparatus of claim 1 wherein said DAC means comprises a single digital 

2 to audio converter having a stereo output comprising first and second output channels, said 

3 first analog audio signals provided to said routing means on said first output channel and 

4 said second analog audio signals provided to said routing means on said second output 

5 channel. 

1 3 . The apparatus of claim 1 wherein said routing means comprises a 

2 multiplexer, said multiplexer controlled by said processor means to selectively couple said 

3 first analog audio signals to said listening device of said first user and said second analog 

4 audio signals to said listening device of said second user. 

1 4 . The apparatus of claim 1 wherein said DAC means comprises a first digital to 

2 audio converter coupled to receive said first digital audio signals and a second digital to audio 

3 converter coupled to receive said second digital audio signals from said sound synthesizer 

4 means. 

1 5 . The apparatus of claim 4 wherein said first and second digital to audio 

2 converters are selectively enabled by said processor means, such that said first digital to 
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3 audio converter is enabled and said second digital to audio converter is disabled when said 

4 sound synthesizer means outputs said first digital audio signals, and said first digital to audio 

5 converter is disabled and said second digital to audio converter is enabled when said sound 

6 synthesizer means outputs said second digital audio signals. 

1 6 . A computer game for providing separate and independent audio to a first and 

2 a second user, comprising: 

3 storage means for storing sound data; 

4 processor means operating under the control of a program and coupled to said 

5 storage means, for selecting and retrieving first sound data for said first user and 

6 second sound data for said second user, 

7 sound synthesizer means coupled to receive said first and said second sound 

8 data and converting said sound data into first and second digital audio signals, 

9 respectively; 

10 controller means coupled to said sound synthesizer to receive said first and 

1 1 second digital audio signals, said controller means further coupled to a bus to provide 

12 said first digital signal to a first user and said second digital sound to a second user, 

13 a first and a second receiving device coupled to said bus for receiving said 

14 first and second digital audio signals, respectively; 

15 conversion means coupled to each of said receiving devices for converting 

16 said received first and second digital audio signals into corresponding first and 

17 second analog audio signals; 

18 a listening device coupled to each of said conversion means for converting 

19 said first and second analog audio signals to sound waves such that said first user 

20 hears said first analog audio signals and said second user hears said second analog 

21 audio signals. 

1 7 . The apparatus of claim 6 wherein said first and second receiving devices 

2 comprise game controllers and said conversion means includes digital to audio converter 

3 (DAC) means for converting said first and second digital audio signals into said 

4 corresponding first and second analog audio signals. 

1 8 . The apparatus of claim 7 wherein each of said game controllers includes input 

2 signal means for receiving input signals from a said user of said game controller, said input 

3 signals including real time sound wherein one said real time sound comprises speech. 

1 9 . The apparatus of claim 8 wherein said bus comprises a serial bus and each of 

2 said game controllers includes a serial bus interface circuit for receiving said first and second 
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3 digital audio signals over said bus and for communicating said input signals from said user 

4 of said game controller to said CPU. 

1 1 0. The apparatus of claim 9 wherein said audio inputs are coupled to said CPU 

2 to be selectively coupled back to said game controllers to provide real time player generated 

3 sounds to said users. 

1 11. A method for providing separate and independent audio to first and second 

2 listeners, comprising the steps of: 

3 providing, substantially under the control of a single executable computer 

4 program, first sound data for a first listener; 

5 providing, substantially under the control of said program, second sound 

6 data for a second listener, said second sound data being different from said first 

7 sound data at some time during the execution of said program; 

8 receiving said first sound data by said first listener; 

9 receiving said second sound data by said second listener; 

10 such that said first and second listeners may be provided with distinct sound data. 

1 1 2. The method of claim 1 1 further comprising the step of providing at least 

2 one of said first or second sound data with an indicator that correlates said first or second 

3 sound data with a corresponding one of said first or second listeners. 

1 1 3 . The method of claim 12 further comprising the step of providing said first 

2 and second sound data in the form of packets and wherein said indicator comprises a 

3 packet address. 

1 14. The method of Claim 1 2 wherein said indicator comprises an "enable" 

2 signal that enables a digital to analog converter associated with one of said first and 

3 second listeners respectively. 

1 15 ethod of claim 1 1 wherein at least one of said first and second 

2 listener . . es a corresponding one of said first or second data in analog form. 

1 16. The method of claim 1 1 wherein said single executable program operates 

2 on a first digital device. 

1 1 7 . The method of claim 1 6 further comprising the step of providing at least 

2 one of said first or second data from said first digital device to a second digital device and 

3 wherein a corresponding one of said first or second listeners receive said first or second 

4 data respectively from said second digital device. 
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1 18. The method of claim 1 7 wherein said first digital device comprises a 

2 server computer and said second digital device comprises a client computer. 

1 1 9 . The method of claim 1 1 wherein said steps of receiving said first and 

2 second data respectively comprise the steps of receiving said first and second data with 

3 first and second multimedia controllers. 

1 20. The method of claim 1 1 wherein said single executable program comprises 

2 a game program. 

1 2 1 . The method of claim 1 1 wherein said single executable program comprises 

2 a computer program controlling an audio program which includes multiple, distinct audio 

3 programs suitable for separate, simultaneous listeners. 

1 22. The method of claim 1 1 wherein said receiving steps comprise receiving 

2 said first or second data respectively in different locations. 

1 23 . The method of claim 1 1 wherein said at least one of first and second 

2 sound data is stored on a mass storage device before being provided to a corresponding 

3 one of said listeners. 

1 24. The method of claim 1 1 further comprising the step of providing said first 

2 and second listeners with visual data associated with said first and second sound data. 

1 25 * The method of claim 1 1 wherein at least one of said steps of providing 

2 said first or second sound data comprises the step of routing said first or second sound 

3 data to a corresponding one of said first or second users. 

1 26. An apparatus for providing separate and independent audio to a first and a 

2 second listener, comprising: 

3 a first digital device including: 

4 first providing means for providing, substantially under the control 

5 of a single executable program, first sound data to a first listener; 

6 second providing means for providing, substantially under the 

7 control of said program, second sound data to a second listener, said 

8 second sound data being different from said first sound data at some time 

9 during the execution of said program; 

10 first receiving means for receiving said first sound data by said first 

1 1 listener; 
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12 second receiving means for receiving said second sound data by said 

1 3 second listener; 

14 such that said first and second listeners may be provided with distinct sound data. 

1 27. The apparatus of claim 26 wherein at least one of said first or second 

2 providing means comprises means for providing said first or second sound data with an 

3 indicator that correlates said first or second sound data with a corresponding one of said 

4 first or second listeners. 

1 28. The apparatus of claim 26 wherein at least one of said firet and second 

2 listeners receives a corresponding one of said first or second data in analog form. 

1 29 . The apparatus of claim 26 further comprising a first digital device and 

2 wherein said single executable program operates on said first digital device. 

1 30. The apparatus of claim 29 further comprising a second digital device and 

2 wherein: 

3 said first or second providing means includes means for providing at least 

4 one of said first or second data from said first digital device to said second digital 

5 device; and 

6 said second digital device includes means for providing said first or 

7 second data from said second digital device to a corresponding one of said first or 

8 second listeners. 

1 31. The apparatus of claim 30 wherein said first digital device comprises a 

2 server computer and said second digital device comprises a client computer. 

1 32. The apparatus of claim 26 wherein said first or second receiving means 

2 comprises a multimedia controller. 

1 33. The apparatus of claim 26 wherein said single executable program 

2 comprises a computer program controlling an audio program which includes multiple, 

3 distinct audio programs suitable for separate, simultaneous listeners. 

1 34. The apparatus of claim 26 wherein said first receiving means comprises a 

2 speaker disposed in a first location and said second receiving means comprises a speaker 

3 disposed in a second location. 

1 35 . The apparatus of claim 26 further comprising a mass storage device for 

2 storing at least one of said first or second sound data before said first or second sound 

3 data is provided to a corresponding one of said first or second listeners. 
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36. The apparatus of claim 26 further comprising means for providing said 
first and second listeners with visual data associated with said first and second sound 
data. 

37 . The apparatus of claim 26 wherein at least one of said first or second 
providing means comprises means for routing said first or second sound data to a 
corresponding one of said first or second users. 

38. A method for providing separate and independent audio to a first and second 
user, comprising the steps of: 

storing sound data; 

selecting and retrieving first sound data for said first user and second sound 
data for said second user; 

receiving said first and second sound data and converting said sound data into 
first and second digital audio signals, respectively; 

receiving said first and second digital audio signals and converting said first 
and second digital audio signals into corresponding first and second analog audio 
signals; 

receiving said first and second analog audio signals and selectively providing 
said first analog audio signals to said first user and said second analog audio signals 
to said second user, 

converting said first and second analog audio signals to sound waves such 
that said first user hears said first analog audio signals and said second user hears 
said second analog audio signals. 

39. A method for providing separate and independent audio to a first and a 
second user, comprising the steps of: 

storing sound data; 

selecting and retrieving first sound data for said first user and second sound 
data for said second user; 

converting said first and second sound data into first and second digital audio 
signals, respectively; 

receiving said first and second digital audio signals, respectively over a bus; 

converting said received first and second digital audio signals into 
corresponding first and second analog audio signals; 

converting said first and second analog audio signals to sound waves such 
that said first user hears said first analog audio signals and said second user hears 
said second analog audio signals. 
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